Beta to Release Matching: Model Selection
Mariana de Oliveira Santos Silva
Last Updated: March 10, 2020
Introduction
The main objective of this notebook is to perform several different matching methods in experiment 3 (which was the experiment that presented the best results in the previous work) to select the best model.
Loading the data
Model Selection
In short, the prior results showed that the model using Nearest Neighbor matching with stratified sampling outperformed the other models. In particular, the experiment 3 (regarding to the results of feature selection with stratified sampling, considering the top 5 features) presented the best results for most of the tested models. However, the results could be further improved. So, before move on to the last phase (the model validation), we can try using other matching strategies. Now that we have selected the best expirement.
Match using Coarsened Exact Matching (CEM): Small Dataset
##
## Using 'treat'='1' as baseline group##
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x_sm,
## method = "cem")
##
## Summary of balance for all data:
## Means Treated Means Control SD Control
## distance 0.6201 0.3793 0.2011
## daily_num_sessions_started 2.8624 2.3903 2.7452
## daily_num_sessions_started_max 5.1994 4.3137 4.8323
## FX_PAGE_LOAD_MS_2_PARENT 3044.1381 3459.4753 1918.2850
## memory_mb 9375.4970 8954.2914 7959.4119
## num_active_days 5.5615 5.3423 2.2659
## num_addons 5.6684 7.8676 3.3668
## num_bookmarks 164.7282 244.0887 1285.4479
## profile_age 899.2399 893.4752 761.9573
## session_length 9.2079 12.2015 14.4045
## session_length_max 18.1451 22.5676 29.8243
## TIME_TO_DOM_COMPLETE_MS 3304.4993 4397.6110 4283.8936
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2291.1852 2743.6285 2737.7315
## TIME_TO_DOM_INTERACTIVE_MS 1797.9580 2411.6625 2431.1910
## TIME_TO_LOAD_EVENT_END_MS 3026.7281 4134.4538 4026.6727
## TIME_TO_NON_BLANK_PAINT_MS 1457.4383 1837.6274 2148.1641
## Mean Diff eQQ Med eQQ Mean eQQ Max
## distance 0.2408 0.2500 0.2408 0.3106
## daily_num_sessions_started 0.4721 0.4000 0.4737 1.5750
## daily_num_sessions_started_max 0.8857 1.0000 0.8882 3.0000
## FX_PAGE_LOAD_MS_2_PARENT -415.3372 286.3527 415.2519 1217.0293
## memory_mb 421.2056 31.0000 542.1348 196547.0000
## num_active_days 0.2192 0.0000 0.2374 1.0000
## num_addons -2.1992 2.0000 2.1994 126.6667
## num_bookmarks -79.3605 1.0000 79.1581 21769.0000
## profile_age 5.7647 23.0000 25.2873 1369.0000
## session_length -2.9936 1.4564 2.9903 151.2126
## session_length_max -4.4224 2.8661 4.4036 858.3503
## TIME_TO_DOM_COMPLETE_MS -1093.1117 408.7792 1092.1439 11538.8571
## TIME_TO_DOM_CONTENT_LOADED_END_MS -452.4433 218.3252 451.4889 12439.4915
## TIME_TO_DOM_INTERACTIVE_MS -613.7044 236.0083 612.6128 20607.0000
## TIME_TO_LOAD_EVENT_END_MS -1107.7257 428.3251 1106.7063 11196.2118
## TIME_TO_NON_BLANK_PAINT_MS -380.1891 151.5375 379.0529 16102.5273
##
##
## Summary of balance for matched data:
## Means Treated Means Control SD Control
## distance 0.6378 0.4615 0.1557
## daily_num_sessions_started 2.2732 2.2302 1.8198
## daily_num_sessions_started_max 4.1538 4.0978 3.3454
## FX_PAGE_LOAD_MS_2_PARENT 2575.2467 2574.2537 1117.0086
## memory_mb 9345.2703 8931.3493 5934.1611
## num_active_days 5.6123 5.6123 2.0231
## num_addons 5.4839 7.0581 1.6139
## num_bookmarks 102.9048 107.0281 266.4720
## profile_age 776.3038 767.3874 682.0415
## session_length 8.9416 9.0935 8.7593
## session_length_max 17.2511 17.3970 16.8420
## TIME_TO_DOM_COMPLETE_MS 2541.2212 2571.6745 1418.3503
## TIME_TO_DOM_CONTENT_LOADED_END_MS 1645.3293 1643.8191 920.1184
## TIME_TO_DOM_INTERACTIVE_MS 1434.9835 1456.7013 775.5078
## TIME_TO_LOAD_EVENT_END_MS 2390.2772 2439.3777 1342.7873
## TIME_TO_NON_BLANK_PAINT_MS 1153.8016 1167.0823 650.7752
## Mean Diff eQQ Med eQQ Mean eQQ Max
## distance 0.1763 0.2110 0.2028 0.2391
## daily_num_sessions_started 0.0430 0.2857 0.2879 2.9583
## daily_num_sessions_started_max 0.0559 0.0000 0.4787 3.0000
## FX_PAGE_LOAD_MS_2_PARENT 0.9930 84.1227 104.1835 676.1569
## memory_mb 413.9210 18.0000 350.8810 16219.0000
## num_active_days 0.0000 0.0000 0.1566 1.0000
## num_addons -1.5742 2.0000 1.8460 4.0000
## num_bookmarks -4.1233 2.0000 15.3790 334.6000
## profile_age 8.9163 47.0000 50.4190 252.0000
## session_length -0.1519 0.8009 1.4499 10.8062
## session_length_max -0.1459 1.8244 1.9894 10.1356
## TIME_TO_DOM_COMPLETE_MS -30.4533 138.3078 187.2241 1005.1325
## TIME_TO_DOM_CONTENT_LOADED_END_MS 1.5102 70.1490 88.3670 1294.2653
## TIME_TO_DOM_INTERACTIVE_MS -21.7178 82.8923 110.1197 870.8117
## TIME_TO_LOAD_EVENT_END_MS -49.1004 156.5560 202.9708 1173.3746
## TIME_TO_NON_BLANK_PAINT_MS -13.2806 54.5940 76.3997 1754.5214
##
## Percent Balance Improvement:
## Mean Diff. eQQ Med eQQ Mean eQQ Max
## distance 26.8023 15.6105 15.7702 23.0370
## daily_num_sessions_started 90.9009 28.5714 39.2304 -87.8307
## daily_num_sessions_started_max 93.6847 100.0000 46.1049 0.0000
## FX_PAGE_LOAD_MS_2_PARENT 99.7609 70.6227 74.9108 44.4420
## memory_mb 1.7295 41.9355 35.2779 91.7480
## num_active_days 100.0000 0.0000 34.0033 0.0000
## num_addons 28.4206 0.0000 16.0687 96.8421
## num_bookmarks 94.8043 -100.0000 80.5718 98.4630
## profile_age -54.6712 -104.3478 -99.3845 81.5924
## session_length 94.9249 45.0069 51.5132 92.8536
## session_length_max 96.7003 36.3442 54.8230 98.8192
## TIME_TO_DOM_COMPLETE_MS 97.2141 66.1656 82.8572 91.2891
## TIME_TO_DOM_CONTENT_LOADED_END_MS 99.6662 67.8695 80.4276 89.5955
## TIME_TO_DOM_INTERACTIVE_MS 96.4612 64.8774 82.0246 95.7742
## TIME_TO_LOAD_EVENT_END_MS 95.5675 63.4493 81.6599 89.5199
## TIME_TO_NON_BLANK_PAINT_MS 96.5068 63.9732 79.8446 89.1041
##
## Sample sizes:
## Control Treated
## All 35027 34973
## Matched 19560 23021
## Unmatched 15467 11952
## Discarded 0 0Match using Coarsened Exact Matching (CEM): Full Dataset
##
## Using 'treat'='1' as baseline groupres_full <- match.data(cem_full)
cem_full_models_1x <- cem_full
cem_full_results_1x <- res_full
print(summary(cem_full))##
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x,
## method = "cem")
##
## Summary of balance for all data:
## Means Treated Means Control SD Control
## distance 0.6213 0.3787 0.2009
## daily_num_sessions_started 2.8876 2.3689 2.7099
## daily_num_sessions_started_max 5.2501 4.2814 4.8046
## FX_PAGE_LOAD_MS_2_PARENT 3027.5228 3463.7084 1920.1256
## memory_mb 9460.0891 8965.1561 7925.6741
## num_active_days 5.5746 5.3462 2.2644
## num_addons 5.6632 7.8554 3.3349
## num_bookmarks 164.1029 242.4878 1292.6145
## profile_age 893.9339 893.7534 762.4791
## session_length 9.2495 12.2962 14.7435
## session_length_max 18.1985 22.7066 30.2272
## TIME_TO_DOM_COMPLETE_MS 3285.4718 4388.9143 4254.1363
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2292.5533 2737.6381 2700.5353
## TIME_TO_DOM_INTERACTIVE_MS 1792.4896 2404.3932 2397.0177
## TIME_TO_LOAD_EVENT_END_MS 3012.8713 4126.9123 4007.0480
## TIME_TO_NON_BLANK_PAINT_MS 1449.9423 1833.6557 2128.4506
## Mean Diff eQQ Med eQQ Mean eQQ Max
## distance 0.2426 0.2527 0.2426 0.3114
## daily_num_sessions_started 0.5188 0.4167 0.5190 1.7500
## daily_num_sessions_started_max 0.9687 1.0000 0.9692 11.0000
## FX_PAGE_LOAD_MS_2_PARENT -436.1856 299.6662 436.2548 1250.9746
## memory_mb 494.9330 31.0000 555.3761 262176.0000
## num_active_days 0.2284 0.0000 0.2413 1.0000
## num_addons -2.1922 2.0000 2.1941 109.0000
## num_bookmarks -78.3849 1.0000 79.0181 21769.0000
## profile_age 0.1805 22.0000 23.1161 1366.0000
## session_length -3.0467 1.4538 3.0489 151.4597
## session_length_max -4.5080 2.7333 4.5187 871.0939
## TIME_TO_DOM_COMPLETE_MS -1103.4424 421.7080 1103.4946 12182.3743
## TIME_TO_DOM_CONTENT_LOADED_END_MS -445.0848 229.0178 445.1832 10634.8539
## TIME_TO_DOM_INTERACTIVE_MS -611.9036 243.7872 612.0582 19062.0000
## TIME_TO_LOAD_EVENT_END_MS -1114.0410 441.3178 1114.0691 11196.2118
## TIME_TO_NON_BLANK_PAINT_MS -383.7134 155.3258 384.1746 19260.1667
##
##
## Summary of balance for matched data:
## Means Treated Means Control SD Control
## distance 0.6368 0.4606 0.1581
## daily_num_sessions_started 2.3597 2.3181 1.9440
## daily_num_sessions_started_max 4.3248 4.2638 3.5848
## FX_PAGE_LOAD_MS_2_PARENT 2630.8495 2629.8182 1165.2841
## memory_mb 9345.9336 9007.7607 6130.1538
## num_active_days 5.6190 5.6190 2.0238
## num_addons 5.4908 7.0621 1.6316
## num_bookmarks 106.6405 108.9676 285.6003
## profile_age 796.3677 786.3890 696.6420
## session_length 9.0017 9.1066 8.9578
## session_length_max 17.4904 17.4516 17.3295
## TIME_TO_DOM_COMPLETE_MS 2610.7773 2636.8150 1491.7543
## TIME_TO_DOM_CONTENT_LOADED_END_MS 1697.3437 1693.6141 997.9611
## TIME_TO_DOM_INTERACTIVE_MS 1469.5869 1489.4234 820.8763
## TIME_TO_LOAD_EVENT_END_MS 2448.9974 2494.9380 1408.4943
## TIME_TO_NON_BLANK_PAINT_MS 1180.7099 1191.0518 681.9404
## Mean Diff eQQ Med eQQ Mean eQQ Max
## distance 0.1762 0.2175 0.2092 0.2479
## daily_num_sessions_started 0.0416 0.3333 0.3373 1.3750
## daily_num_sessions_started_max 0.0610 1.0000 0.5791 3.0000
## FX_PAGE_LOAD_MS_2_PARENT 1.0313 88.2880 115.8262 1222.0000
## memory_mb 338.1729 14.0000 309.7014 16281.0000
## num_active_days 0.0000 0.0000 0.1516 1.0000
## num_addons -1.5713 2.0000 1.9054 4.8333
## num_bookmarks -2.3271 2.0000 17.9224 2354.8750
## profile_age 9.9787 36.0000 43.8503 246.0000
## session_length -0.1049 0.9890 1.5868 6.6953
## session_length_max 0.0388 1.9725 2.1093 30.2350
## TIME_TO_DOM_COMPLETE_MS -26.0378 139.8803 200.9518 1109.0728
## TIME_TO_DOM_CONTENT_LOADED_END_MS 3.7296 73.2474 96.4370 1088.3989
## TIME_TO_DOM_INTERACTIVE_MS -19.8365 85.7666 120.1839 1005.4136
## TIME_TO_LOAD_EVENT_END_MS -45.9406 159.2815 221.6872 3788.0222
## TIME_TO_NON_BLANK_PAINT_MS -10.3419 55.7787 79.7257 1161.2917
##
## Percent Balance Improvement:
## Mean Diff. eQQ Med eQQ Mean eQQ Max
## distance 27.3811 13.9260 13.7551 20.3929
## daily_num_sessions_started 91.9833 20.0000 35.0104 21.4286
## daily_num_sessions_started_max 93.7068 0.0000 40.2527 72.7273
## FX_PAGE_LOAD_MS_2_PARENT 99.7636 70.5379 73.4499 2.3162
## memory_mb 31.6730 54.8387 44.2357 93.7900
## num_active_days 100.0000 0.0000 37.1827 0.0000
## num_addons 28.3232 0.0000 13.1586 95.5657
## num_bookmarks 97.0312 -100.0000 77.3186 89.1824
## profile_age -5427.1469 -63.6364 -89.6960 81.9912
## session_length 96.5556 31.9662 47.9543 95.5795
## session_length_max 99.1398 27.8354 53.3211 96.5291
## TIME_TO_DOM_COMPLETE_MS 97.6403 66.8301 81.7895 90.8961
## TIME_TO_DOM_CONTENT_LOADED_END_MS 99.1620 68.0167 78.3377 89.7657
## TIME_TO_DOM_INTERACTIVE_MS 96.7582 64.8191 80.3640 94.7256
## TIME_TO_LOAD_EVENT_END_MS 95.8762 63.9078 80.1011 66.1669
## TIME_TO_NON_BLANK_PAINT_MS 97.3048 64.0892 79.2475 93.9705
##
## Sample sizes:
## Control Treated
## All 59627 59627
## Matched 36765 42779
## Unmatched 22862 16848
## Discarded 0 0Match using Nearest Neighbor matching: Small Dataset
nn <- matchit(formula = generate_formula(exp_3, label), df_1x_sm, 'nearest', replace = TRUE)
res <- match.data(nn)
nn_models_1x <- nn
nn_results_1x <- res
print(summary(nn))##
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x_sm,
## method = "nearest", replace = TRUE)
##
## Summary of balance for all data:
## Means Treated Means Control SD Control
## distance 0.6201 0.3793 0.2011
## daily_num_sessions_started 2.8624 2.3903 2.7452
## daily_num_sessions_started_max 5.1994 4.3137 4.8323
## FX_PAGE_LOAD_MS_2_PARENT 3044.1381 3459.4753 1918.2850
## memory_mb 9375.4970 8954.2914 7959.4119
## num_active_days 5.5615 5.3423 2.2659
## num_addons 5.6684 7.8676 3.3668
## num_bookmarks 164.7282 244.0887 1285.4479
## profile_age 899.2399 893.4752 761.9573
## session_length 9.2079 12.2015 14.4045
## session_length_max 18.1451 22.5676 29.8243
## TIME_TO_DOM_COMPLETE_MS 3304.4993 4397.6110 4283.8936
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2291.1852 2743.6285 2737.7315
## TIME_TO_DOM_INTERACTIVE_MS 1797.9580 2411.6625 2431.1910
## TIME_TO_LOAD_EVENT_END_MS 3026.7281 4134.4538 4026.6727
## TIME_TO_NON_BLANK_PAINT_MS 1457.4383 1837.6274 2148.1641
## Mean Diff eQQ Med eQQ Mean eQQ Max
## distance 0.2408 0.2500 0.2408 0.3106
## daily_num_sessions_started 0.4721 0.4000 0.4737 1.5750
## daily_num_sessions_started_max 0.8857 1.0000 0.8882 3.0000
## FX_PAGE_LOAD_MS_2_PARENT -415.3372 286.3527 415.2519 1217.0293
## memory_mb 421.2056 31.0000 542.1348 196547.0000
## num_active_days 0.2192 0.0000 0.2374 1.0000
## num_addons -2.1992 2.0000 2.1994 126.6667
## num_bookmarks -79.3605 1.0000 79.1581 21769.0000
## profile_age 5.7647 23.0000 25.2873 1369.0000
## session_length -2.9936 1.4564 2.9903 151.2126
## session_length_max -4.4224 2.8661 4.4036 858.3503
## TIME_TO_DOM_COMPLETE_MS -1093.1117 408.7792 1092.1439 11538.8571
## TIME_TO_DOM_CONTENT_LOADED_END_MS -452.4433 218.3252 451.4889 12439.4915
## TIME_TO_DOM_INTERACTIVE_MS -613.7044 236.0083 612.6128 20607.0000
## TIME_TO_LOAD_EVENT_END_MS -1107.7257 428.3251 1106.7063 11196.2118
## TIME_TO_NON_BLANK_PAINT_MS -380.1891 151.5375 379.0529 16102.5273
##
##
## Summary of balance for matched data:
## Means Treated Means Control SD Control
## distance 0.6201 0.6201 0.1995
## daily_num_sessions_started 2.8624 4.0732 4.9384
## daily_num_sessions_started_max 5.1994 7.6838 9.8565
## FX_PAGE_LOAD_MS_2_PARENT 3044.1381 2912.7561 1622.9078
## memory_mb 9375.4970 14598.4850 18130.7604
## num_active_days 5.5615 5.9370 2.0637
## num_addons 5.6684 6.2446 1.9905
## num_bookmarks 164.7282 851.4996 3905.5180
## profile_age 899.2399 949.5171 809.2864
## session_length 9.2079 8.1689 10.8524
## session_length_max 18.1451 17.1514 39.7136
## TIME_TO_DOM_COMPLETE_MS 3304.4993 3195.2355 3352.6550
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2291.1852 2805.0239 3838.0612
## TIME_TO_DOM_INTERACTIVE_MS 1797.9580 1726.9272 1763.8950
## TIME_TO_LOAD_EVENT_END_MS 3026.7281 2854.4041 2673.1022
## TIME_TO_NON_BLANK_PAINT_MS 1457.4383 1397.7523 1622.3912
## Mean Diff eQQ Med eQQ Mean eQQ Max
## distance 0.0000 0.1645 0.1589 0.2015
## daily_num_sessions_started -1.2108 0.2321 0.2508 2.8750
## daily_num_sessions_started_max -2.4844 0.0000 0.4174 9.0000
## FX_PAGE_LOAD_MS_2_PARENT 131.3820 119.0923 208.8125 912.0536
## memory_mb -5222.9880 29.0000 727.7257 196581.0000
## num_active_days -0.3755 0.0000 0.1748 1.0000
## num_addons -0.5762 1.5000 1.4821 11.0000
## num_bookmarks -686.7714 2.0000 81.0448 21802.0000
## profile_age -50.2772 13.0000 19.4053 1597.0000
## session_length 1.0390 0.2017 1.4697 144.9158
## session_length_max 0.9938 0.6333 2.3059 858.3503
## TIME_TO_DOM_COMPLETE_MS 109.2638 147.9601 562.4198 11171.1250
## TIME_TO_DOM_CONTENT_LOADED_END_MS -513.8387 88.5859 298.0847 12439.4915
## TIME_TO_DOM_INTERACTIVE_MS 71.0308 78.5130 318.7461 20607.0000
## TIME_TO_LOAD_EVENT_END_MS 172.3240 147.1974 563.7810 7937.8281
## TIME_TO_NON_BLANK_PAINT_MS 59.6860 56.2857 202.8511 9919.0000
##
## Percent Balance Improvement:
## Mean Diff. eQQ Med eQQ Mean eQQ Max
## distance 99.9987 34.2278 34.0267 35.1400
## daily_num_sessions_started -156.4632 41.9643 47.0445 -82.5397
## daily_num_sessions_started_max -180.5042 100.0000 53.0022 -200.0000
## FX_PAGE_LOAD_MS_2_PARENT 68.3674 58.4106 49.7142 25.0590
## memory_mb -1140.0092 6.4516 -34.2334 -0.0173
## num_active_days -71.2910 0.0000 26.3693 0.0000
## num_addons 73.7992 25.0000 32.6121 91.3158
## num_bookmarks -765.3816 -100.0000 -2.3834 -0.1516
## profile_age -772.1575 43.4783 23.2609 -16.6545
## session_length 65.2940 86.1510 50.8513 4.1642
## session_length_max 77.5289 77.9027 47.6370 0.0000
## TIME_TO_DOM_COMPLETE_MS 90.0043 63.8044 48.5031 3.1869
## TIME_TO_DOM_CONTENT_LOADED_END_MS -13.5698 59.4248 33.9774 0.0000
## TIME_TO_DOM_INTERACTIVE_MS 88.4259 66.7330 47.9694 0.0000
## TIME_TO_LOAD_EVENT_END_MS 84.4434 65.6342 49.0578 29.1026
## TIME_TO_NON_BLANK_PAINT_MS 84.3010 62.8569 46.4848 38.4010
##
## Sample sizes:
## Control Treated
## All 35027 34973
## Matched 12491 34973
## Unmatched 22536 0
## Discarded 0 0Match using Nearest Neighbor matching: Full Dataset
nn_full <- matchit(formula = generate_formula(exp_3, label), df_1x, 'nearest', replace = TRUE)
res <- match.data(nn_full)
nn_full_models_1x <- nn_full
nn_full_results_1x <- res
print(summary(nn_full))##
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x,
## method = "nearest", replace = TRUE)
##
## Summary of balance for all data:
## Means Treated Means Control SD Control
## distance 0.6213 0.3787 0.2009
## daily_num_sessions_started 2.8876 2.3689 2.7099
## daily_num_sessions_started_max 5.2501 4.2814 4.8046
## FX_PAGE_LOAD_MS_2_PARENT 3027.5228 3463.7084 1920.1256
## memory_mb 9460.0891 8965.1561 7925.6741
## num_active_days 5.5746 5.3462 2.2644
## num_addons 5.6632 7.8554 3.3349
## num_bookmarks 164.1029 242.4878 1292.6145
## profile_age 893.9339 893.7534 762.4791
## session_length 9.2495 12.2962 14.7435
## session_length_max 18.1985 22.7066 30.2272
## TIME_TO_DOM_COMPLETE_MS 3285.4718 4388.9143 4254.1363
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2292.5533 2737.6381 2700.5353
## TIME_TO_DOM_INTERACTIVE_MS 1792.4896 2404.3932 2397.0177
## TIME_TO_LOAD_EVENT_END_MS 3012.8713 4126.9123 4007.0480
## TIME_TO_NON_BLANK_PAINT_MS 1449.9423 1833.6557 2128.4506
## Mean Diff eQQ Med eQQ Mean eQQ Max
## distance 0.2426 0.2527 0.2426 0.3114
## daily_num_sessions_started 0.5188 0.4167 0.5190 1.7500
## daily_num_sessions_started_max 0.9687 1.0000 0.9692 11.0000
## FX_PAGE_LOAD_MS_2_PARENT -436.1856 299.6662 436.2548 1250.9746
## memory_mb 494.9330 31.0000 555.3761 262176.0000
## num_active_days 0.2284 0.0000 0.2413 1.0000
## num_addons -2.1922 2.0000 2.1941 109.0000
## num_bookmarks -78.3849 1.0000 79.0181 21769.0000
## profile_age 0.1805 22.0000 23.1161 1366.0000
## session_length -3.0467 1.4538 3.0489 151.4597
## session_length_max -4.5080 2.7333 4.5187 871.0939
## TIME_TO_DOM_COMPLETE_MS -1103.4424 421.7080 1103.4946 12182.3743
## TIME_TO_DOM_CONTENT_LOADED_END_MS -445.0848 229.0178 445.1832 10634.8539
## TIME_TO_DOM_INTERACTIVE_MS -611.9036 243.7872 612.0582 19062.0000
## TIME_TO_LOAD_EVENT_END_MS -1114.0410 441.3178 1114.0691 11196.2118
## TIME_TO_NON_BLANK_PAINT_MS -383.7134 155.3258 384.1746 19260.1667
##
##
## Summary of balance for matched data:
## Means Treated Means Control SD Control
## distance 0.6213 0.6213 0.2000
## daily_num_sessions_started 2.8876 4.3582 5.2354
## daily_num_sessions_started_max 5.2501 8.0823 9.5927
## FX_PAGE_LOAD_MS_2_PARENT 3027.5228 2894.2370 1635.9350
## memory_mb 9460.0891 14479.9604 18183.1455
## num_active_days 5.5746 5.9842 2.0373
## num_addons 5.6632 6.2566 2.0935
## num_bookmarks 164.1029 603.6715 3468.5644
## profile_age 893.9339 921.9102 805.7037
## session_length 9.2495 8.2741 12.0331
## session_length_max 18.1985 17.9987 50.5116
## TIME_TO_DOM_COMPLETE_MS 3285.4718 3128.3417 3058.5441
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2292.5533 2820.1932 3950.7895
## TIME_TO_DOM_INTERACTIVE_MS 1792.4896 1712.4133 1782.8317
## TIME_TO_LOAD_EVENT_END_MS 3012.8713 2816.9697 2652.6928
## TIME_TO_NON_BLANK_PAINT_MS 1449.9423 1410.9626 1885.5897
## Mean Diff eQQ Med eQQ Mean eQQ Max
## distance 0.0000 0.1655 0.1594 0.2022
## daily_num_sessions_started -1.4706 0.2083 0.2345 2.4643
## daily_num_sessions_started_max -2.8322 0.0000 0.4117 10.0000
## FX_PAGE_LOAD_MS_2_PARENT 133.2858 119.5299 219.7080 977.6778
## memory_mb -5019.8713 27.0000 682.6319 262176.0000
## num_active_days -0.4096 0.0000 0.1577 1.0000
## num_addons -0.5934 1.5000 1.4923 109.0000
## num_bookmarks -439.5686 1.6667 79.1675 21761.6250
## profile_age -27.9763 21.0000 25.2114 1231.0000
## session_length 0.9754 0.1928 1.3577 140.8883
## session_length_max 0.1998 0.5817 2.1986 763.0081
## TIME_TO_DOM_COMPLETE_MS 157.1302 160.5541 573.2693 11538.8571
## TIME_TO_DOM_CONTENT_LOADED_END_MS -527.6399 92.3230 284.1487 10634.8539
## TIME_TO_DOM_INTERACTIVE_MS 80.0763 88.6948 324.3294 18812.3581
## TIME_TO_LOAD_EVENT_END_MS 195.9016 158.7490 568.8458 11171.1250
## TIME_TO_NON_BLANK_PAINT_MS 38.9797 61.6206 212.4963 17714.0000
##
## Percent Balance Improvement:
## Mean Diff. eQQ Med eQQ Mean eQQ Max
## distance 99.9995 34.5097 34.2993 35.0577
## daily_num_sessions_started -183.4842 50.0000 54.8258 -40.8163
## daily_num_sessions_started_max -192.3720 100.0000 57.5216 9.0909
## FX_PAGE_LOAD_MS_2_PARENT 69.4429 60.1123 49.6377 21.8467
## memory_mb -914.2527 12.9032 -22.9134 0.0000
## num_active_days -79.3392 0.0000 34.6539 0.0000
## num_addons 72.9305 25.0000 31.9864 0.0000
## num_bookmarks -460.7820 -66.6667 -0.1891 0.0339
## profile_age -15395.9870 4.5455 -9.0644 9.8829
## session_length 67.9842 86.7345 55.4693 6.9797
## session_length_max 95.5681 78.7196 51.3448 12.4081
## TIME_TO_DOM_COMPLETE_MS 85.7600 61.9277 48.0496 5.2824
## TIME_TO_DOM_CONTENT_LOADED_END_MS -18.5482 59.6874 36.1726 0.0000
## TIME_TO_DOM_INTERACTIVE_MS 86.9136 63.6180 47.0100 1.3096
## TIME_TO_LOAD_EVENT_END_MS 82.4152 64.0284 48.9398 0.2241
## TIME_TO_NON_BLANK_PAINT_MS 89.8414 60.3282 44.6876 8.0278
##
## Sample sizes:
## Control Treated
## All 59627 59627
## Matched 21147 59627
## Unmatched 38480 0
## Discarded 0 0Match using Nearest Neighbor matching: Small Dataset - Mahalanobis
nn.ma <- matchit(formula = generate_formula(exp_3, label), df_1x_sm, 'nearest', distance='mahalanobis')
res <- match.data(nn.ma)
nn.ma_models_1x <- nn.ma
nn.ma_results_1x <- res
print(summary(nn.ma))##
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x_sm,
## method = "nearest", distance = "mahalanobis")
##
## Summary of balance for all data:
## Means Treated Means Control SD Control
## daily_num_sessions_started 2.8624 2.3903 2.7452
## daily_num_sessions_started_max 5.1994 4.3137 4.8323
## FX_PAGE_LOAD_MS_2_PARENT 3044.1381 3459.4753 1918.2850
## memory_mb 9375.4970 8954.2914 7959.4119
## num_active_days 5.5615 5.3423 2.2659
## num_addons 5.6684 7.8676 3.3668
## num_bookmarks 164.7282 244.0887 1285.4479
## profile_age 899.2399 893.4752 761.9573
## session_length 9.2079 12.2015 14.4045
## session_length_max 18.1451 22.5676 29.8243
## TIME_TO_DOM_COMPLETE_MS 3304.4993 4397.6110 4283.8936
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2291.1852 2743.6285 2737.7315
## TIME_TO_DOM_INTERACTIVE_MS 1797.9580 2411.6625 2431.1910
## TIME_TO_LOAD_EVENT_END_MS 3026.7281 4134.4538 4026.6727
## TIME_TO_NON_BLANK_PAINT_MS 1457.4383 1837.6274 2148.1641
## Mean Diff eQQ Med eQQ Mean eQQ Max
## daily_num_sessions_started 0.4721 0.4000 0.4737 1.5750
## daily_num_sessions_started_max 0.8857 1.0000 0.8882 3.0000
## FX_PAGE_LOAD_MS_2_PARENT -415.3372 286.3527 415.2519 1217.0293
## memory_mb 421.2056 31.0000 542.1348 196547.0000
## num_active_days 0.2192 0.0000 0.2374 1.0000
## num_addons -2.1992 2.0000 2.1994 126.6667
## num_bookmarks -79.3605 1.0000 79.1581 21769.0000
## profile_age 5.7647 23.0000 25.2873 1369.0000
## session_length -2.9936 1.4564 2.9903 151.2126
## session_length_max -4.4224 2.8661 4.4036 858.3503
## TIME_TO_DOM_COMPLETE_MS -1093.1117 408.7792 1092.1439 11538.8571
## TIME_TO_DOM_CONTENT_LOADED_END_MS -452.4433 218.3252 451.4889 12439.4915
## TIME_TO_DOM_INTERACTIVE_MS -613.7044 236.0083 612.6128 20607.0000
## TIME_TO_LOAD_EVENT_END_MS -1107.7257 428.3251 1106.7063 11196.2118
## TIME_TO_NON_BLANK_PAINT_MS -380.1891 151.5375 379.0529 16102.5273
##
##
## Summary of balance for matched data:
## Means Treated Means Control SD Control
## daily_num_sessions_started 2.8624 2.3903 2.7451
## daily_num_sessions_started_max 5.1994 4.3144 4.8332
## FX_PAGE_LOAD_MS_2_PARENT 3044.1381 3456.7463 1914.4418
## memory_mb 9375.4970 8950.7645 7959.4384
## num_active_days 5.5615 5.3435 2.2649
## num_addons 5.6684 7.8592 3.2436
## num_bookmarks 164.7282 221.2422 987.5952
## profile_age 899.2399 893.6282 762.0575
## session_length 9.2079 12.1616 14.1624
## session_length_max 18.1451 22.3865 26.9638
## TIME_TO_DOM_COMPLETE_MS 3304.4993 4380.9954 4213.0576
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2291.1852 2732.9433 2676.6454
## TIME_TO_DOM_INTERACTIVE_MS 1797.9580 2400.7730 2358.5067
## TIME_TO_LOAD_EVENT_END_MS 3026.7281 4121.5844 3975.2591
## TIME_TO_NON_BLANK_PAINT_MS 1457.4383 1824.7957 1989.4600
## Mean Diff eQQ Med eQQ Mean eQQ Max
## daily_num_sessions_started 0.4721 0.4000 0.4734 1.6250
## daily_num_sessions_started_max 0.8850 1.0000 0.8868 4.0000
## FX_PAGE_LOAD_MS_2_PARENT -412.6082 285.7252 412.6591 1206.4978
## memory_mb 424.7325 31.0000 541.6251 196547.0000
## num_active_days 0.2180 0.0000 0.2365 1.0000
## num_addons -2.1908 2.0000 2.1925 26.6667
## num_bookmarks -56.5140 1.0000 57.3093 4825.5000
## profile_age 5.6118 23.0000 25.2124 1385.0000
## session_length -2.9538 1.4542 2.9556 151.2126
## session_length_max -4.2413 2.8572 4.2510 174.1819
## TIME_TO_DOM_COMPLETE_MS -1076.4961 408.0023 1076.5944 8525.6667
## TIME_TO_DOM_CONTENT_LOADED_END_MS -441.7581 217.8830 441.8767 5888.6348
## TIME_TO_DOM_INTERACTIVE_MS -602.8150 235.7796 602.8969 6066.2727
## TIME_TO_LOAD_EVENT_END_MS -1094.8563 427.5796 1094.8774 8904.8009
## TIME_TO_NON_BLANK_PAINT_MS -367.3575 151.2618 370.2076 26903.0000
##
## Percent Balance Improvement:
## Mean Diff. eQQ Med eQQ Mean eQQ Max
## daily_num_sessions_started -0.0029 0.0000 0.0597 -3.1746
## daily_num_sessions_started_max 0.0805 0.0000 0.1545 -33.3333
## FX_PAGE_LOAD_MS_2_PARENT 0.6571 0.2191 0.6244 0.8653
## memory_mb -0.8373 0.0000 0.0940 0.0000
## num_active_days 0.5411 0.0000 0.3614 0.0000
## num_addons 0.3846 0.0000 0.3148 78.9474
## num_bookmarks 28.7882 0.0000 27.6015 77.8332
## profile_age 2.6530 0.0000 0.2964 -1.1687
## session_length 1.3320 0.1537 1.1626 0.0000
## session_length_max 4.0949 0.3101 3.4646 79.7074
## TIME_TO_DOM_COMPLETE_MS 1.5200 0.1900 1.4238 26.1134
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2.3617 0.2025 2.1290 52.6618
## TIME_TO_DOM_INTERACTIVE_MS 1.7744 0.0969 1.5860 70.5621
## TIME_TO_LOAD_EVENT_END_MS 1.1618 0.1740 1.0688 20.4659
## TIME_TO_NON_BLANK_PAINT_MS 3.3751 0.1819 2.3335 -67.0732
##
## Sample sizes:
## Control Treated
## All 35027 34973
## Matched 34973 34973
## Unmatched 54 0
## Discarded 0 0Match using Nearest Neighbor matching: Full Dataset - Mahalanobis
nn.ma_full <- matchit(formula = generate_formula(exp_3, label), df_1x, 'nearest', distance='mahalanobis')
res <- match.data(nn.ma_full)
nn.ma_full_models_1x <- nn.ma_full
nn.ma_full_results_1x <- res
print(summary(nn.ma_full))##
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x,
## method = "nearest", distance = "mahalanobis")
##
## Summary of balance for all data:
## Means Treated Means Control SD Control
## daily_num_sessions_started 2.8876 2.3689 2.7099
## daily_num_sessions_started_max 5.2501 4.2814 4.8046
## FX_PAGE_LOAD_MS_2_PARENT 3027.5228 3463.7084 1920.1256
## memory_mb 9460.0891 8965.1561 7925.6741
## num_active_days 5.5746 5.3462 2.2644
## num_addons 5.6632 7.8554 3.3349
## num_bookmarks 164.1029 242.4878 1292.6145
## profile_age 893.9339 893.7534 762.4791
## session_length 9.2495 12.2962 14.7435
## session_length_max 18.1985 22.7066 30.2272
## TIME_TO_DOM_COMPLETE_MS 3285.4718 4388.9143 4254.1363
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2292.5533 2737.6381 2700.5353
## TIME_TO_DOM_INTERACTIVE_MS 1792.4896 2404.3932 2397.0177
## TIME_TO_LOAD_EVENT_END_MS 3012.8713 4126.9123 4007.0480
## TIME_TO_NON_BLANK_PAINT_MS 1449.9423 1833.6557 2128.4506
## Mean Diff eQQ Med eQQ Mean eQQ Max
## daily_num_sessions_started 0.5188 0.4167 0.5190 1.7500
## daily_num_sessions_started_max 0.9687 1.0000 0.9692 11.0000
## FX_PAGE_LOAD_MS_2_PARENT -436.1856 299.6662 436.2548 1250.9746
## memory_mb 494.9330 31.0000 555.3761 262176.0000
## num_active_days 0.2284 0.0000 0.2413 1.0000
## num_addons -2.1922 2.0000 2.1941 109.0000
## num_bookmarks -78.3849 1.0000 79.0181 21769.0000
## profile_age 0.1805 22.0000 23.1161 1366.0000
## session_length -3.0467 1.4538 3.0489 151.4597
## session_length_max -4.5080 2.7333 4.5187 871.0939
## TIME_TO_DOM_COMPLETE_MS -1103.4424 421.7080 1103.4946 12182.3743
## TIME_TO_DOM_CONTENT_LOADED_END_MS -445.0848 229.0178 445.1832 10634.8539
## TIME_TO_DOM_INTERACTIVE_MS -611.9036 243.7872 612.0582 19062.0000
## TIME_TO_LOAD_EVENT_END_MS -1114.0410 441.3178 1114.0691 11196.2118
## TIME_TO_NON_BLANK_PAINT_MS -383.7134 155.3258 384.1746 19260.1667
##
##
## Summary of balance for matched data:
## Means Treated Means Control SD Control
## daily_num_sessions_started 2.8876 2.3689 2.7099
## daily_num_sessions_started_max 5.2501 4.2814 4.8046
## FX_PAGE_LOAD_MS_2_PARENT 3027.5228 3463.7084 1920.1256
## memory_mb 9460.0891 8965.1561 7925.6741
## num_active_days 5.5746 5.3462 2.2644
## num_addons 5.6632 7.8554 3.3349
## num_bookmarks 164.1029 242.4878 1292.6145
## profile_age 893.9339 893.7534 762.4791
## session_length 9.2495 12.2962 14.7435
## session_length_max 18.1985 22.7066 30.2272
## TIME_TO_DOM_COMPLETE_MS 3285.4718 4388.9143 4254.1363
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2292.5533 2737.6381 2700.5353
## TIME_TO_DOM_INTERACTIVE_MS 1792.4896 2404.3932 2397.0177
## TIME_TO_LOAD_EVENT_END_MS 3012.8713 4126.9123 4007.0480
## TIME_TO_NON_BLANK_PAINT_MS 1449.9423 1833.6557 2128.4506
## Mean Diff eQQ Med eQQ Mean eQQ Max
## daily_num_sessions_started 0.5188 0.4167 0.5190 1.7500
## daily_num_sessions_started_max 0.9687 1.0000 0.9692 11.0000
## FX_PAGE_LOAD_MS_2_PARENT -436.1856 299.6662 436.2548 1250.9746
## memory_mb 494.9330 31.0000 555.3761 262176.0000
## num_active_days 0.2284 0.0000 0.2413 1.0000
## num_addons -2.1922 2.0000 2.1941 109.0000
## num_bookmarks -78.3849 1.0000 79.0181 21769.0000
## profile_age 0.1805 22.0000 23.1161 1366.0000
## session_length -3.0467 1.4538 3.0489 151.4597
## session_length_max -4.5080 2.7333 4.5187 871.0939
## TIME_TO_DOM_COMPLETE_MS -1103.4424 421.7080 1103.4946 12182.3743
## TIME_TO_DOM_CONTENT_LOADED_END_MS -445.0848 229.0178 445.1832 10634.8539
## TIME_TO_DOM_INTERACTIVE_MS -611.9036 243.7872 612.0582 19062.0000
## TIME_TO_LOAD_EVENT_END_MS -1114.0410 441.3178 1114.0691 11196.2118
## TIME_TO_NON_BLANK_PAINT_MS -383.7134 155.3258 384.1746 19260.1667
##
## Percent Balance Improvement:
## Mean Diff. eQQ Med eQQ Mean eQQ Max
## daily_num_sessions_started 0 0 0 0
## daily_num_sessions_started_max 0 0 0 0
## FX_PAGE_LOAD_MS_2_PARENT 0 0 0 0
## memory_mb 0 0 0 0
## num_active_days 0 0 0 0
## num_addons 0 0 0 0
## num_bookmarks 0 0 0 0
## profile_age 0 0 0 0
## session_length 0 0 0 0
## session_length_max 0 0 0 0
## TIME_TO_DOM_COMPLETE_MS 0 0 0 0
## TIME_TO_DOM_CONTENT_LOADED_END_MS 0 0 0 0
## TIME_TO_DOM_INTERACTIVE_MS 0 0 0 0
## TIME_TO_LOAD_EVENT_END_MS 0 0 0 0
## TIME_TO_NON_BLANK_PAINT_MS 0 0 0 0
##
## Sample sizes:
## Control Treated
## All 59627 59627
## Matched 59627 59627
## Unmatched 0 0
## Discarded 0 0Match using Subclassification matching: Small Dataset
sub <- matchit(formula = generate_formula(exp_3, label), df_1x_sm, 'subclass')
res <- match.data(sub)
sub_models_1x <- sub
sub_results_1x <- res
print(summary(sub))##
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x_sm,
## method = "subclass", sub.by = "treat")
## Summary of balance for all data:
## Means Treated Means Control Mean Diff
## distance 0.6201 0.3793 0.2408
## daily_num_sessions_started 2.8624 2.3903 0.4721
## daily_num_sessions_started_max 5.1994 4.3137 0.8857
## FX_PAGE_LOAD_MS_2_PARENT 3044.1381 3459.4753 -415.3372
## memory_mb 9375.4970 8954.2914 421.2056
## num_active_days 5.5615 5.3423 0.2192
## num_addons 5.6684 7.8676 -2.1992
## num_bookmarks 164.7282 244.0887 -79.3605
## profile_age 899.2399 893.4752 5.7647
## session_length 9.2079 12.2015 -2.9936
## session_length_max 18.1451 22.5676 -4.4224
## TIME_TO_DOM_COMPLETE_MS 3304.4993 4397.6110 -1093.1117
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2291.1852 2743.6285 -452.4433
## TIME_TO_DOM_INTERACTIVE_MS 1797.9580 2411.6625 -613.7044
## TIME_TO_LOAD_EVENT_END_MS 3026.7281 4134.4538 -1107.7257
## TIME_TO_NON_BLANK_PAINT_MS 1457.4383 1837.6274 -380.1891
## eQQ Med eQQ Mean eQQ Max
## distance 0.2500 0.2408 0.3106
## daily_num_sessions_started 0.4000 0.4737 1.5750
## daily_num_sessions_started_max 1.0000 0.8882 3.0000
## FX_PAGE_LOAD_MS_2_PARENT 286.3527 415.2519 1217.0293
## memory_mb 31.0000 542.1348 196547.0000
## num_active_days 0.0000 0.2374 1.0000
## num_addons 2.0000 2.1994 126.6667
## num_bookmarks 1.0000 79.1581 21769.0000
## profile_age 23.0000 25.2873 1369.0000
## session_length 1.4564 2.9903 151.2126
## session_length_max 2.8661 4.4036 858.3503
## TIME_TO_DOM_COMPLETE_MS 408.7792 1092.1439 11538.8571
## TIME_TO_DOM_CONTENT_LOADED_END_MS 218.3252 451.4889 12439.4915
## TIME_TO_DOM_INTERACTIVE_MS 236.0083 612.6128 20607.0000
## TIME_TO_LOAD_EVENT_END_MS 428.3251 1106.7063 11196.2118
## TIME_TO_NON_BLANK_PAINT_MS 151.5375 379.0529 16102.5273
##
##
## Summary of balance by subclasses:
## , , Subclass 1
##
## Means Treated Means Control Mean Diff
## distance 0.2509 0.2285 0.0225
## daily_num_sessions_started 2.3990 2.0658 0.3332
## daily_num_sessions_started_max 4.4088 3.6718 0.7371
## FX_PAGE_LOAD_MS_2_PARENT 3629.1937 3886.2732 -257.0795
## memory_mb 9138.1626 8798.5694 339.5932
## num_active_days 5.5323 5.0662 0.4662
## num_addons 9.1710 9.2452 -0.0743
## num_bookmarks 300.3797 295.9452 4.4345
## profile_age 940.3483 867.1655 73.1827
## session_length 12.4357 15.0351 -2.5994
## session_length_max 23.2722 26.7248 -3.4525
## TIME_TO_DOM_COMPLETE_MS 4655.6138 5529.7289 -874.1151
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2943.9156 3218.5432 -274.6276
## TIME_TO_DOM_INTERACTIVE_MS 2547.7871 3027.4856 -479.6985
## TIME_TO_LOAD_EVENT_END_MS 4370.7806 5268.1085 -897.3279
## TIME_TO_NON_BLANK_PAINT_MS 1986.8516 2213.4394 -226.5878
## eQQ Med eQQ Mean eQQ Max
## distance 0.0230 0.0225 0.0412
## daily_num_sessions_started 0.3333 0.3565 5.7500
## daily_num_sessions_started_max 1.0000 0.7739 30.0000
## FX_PAGE_LOAD_MS_2_PARENT 188.9938 280.2886 727.4286
## memory_mb 40.0000 616.1848 32765.0000
## num_active_days 0.0000 0.4665 1.0000
## num_addons 0.3333 0.4939 126.6667
## num_bookmarks 22.0000 53.0462 22426.0000
## profile_age 78.0000 74.9695 1231.0000
## session_length 1.7838 2.6090 151.2126
## session_length_max 1.7144 3.5516 586.3783
## TIME_TO_DOM_COMPLETE_MS 396.3856 888.4752 6719.6488
## TIME_TO_DOM_CONTENT_LOADED_END_MS 131.1098 280.2402 19981.4000
## TIME_TO_DOM_INTERACTIVE_MS 195.7931 485.9224 22814.3333
## TIME_TO_LOAD_EVENT_END_MS 412.7270 907.7385 6719.6488
## TIME_TO_NON_BLANK_PAINT_MS 65.0801 239.1967 49949.1429
##
## , , Subclass 2
##
## Means Treated Means Control Mean Diff
## distance 0.5258 0.5173 0.0085
## daily_num_sessions_started 2.4010 2.3259 0.0751
## daily_num_sessions_started_max 4.3050 4.1744 0.1307
## FX_PAGE_LOAD_MS_2_PARENT 3358.1953 3056.1402 302.0551
## memory_mb 8490.5660 8053.3879 437.1781
## num_active_days 5.2675 5.4117 -0.1443
## num_addons 6.1513 6.3617 -0.2104
## num_bookmarks 174.3964 125.2153 49.1810
## profile_age 929.3821 892.7549 36.6271
## session_length 11.2572 9.8249 1.4323
## session_length_max 21.6842 18.9055 2.7787
## TIME_TO_DOM_COMPLETE_MS 3838.2941 3220.1870 618.1071
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2504.7020 2131.5823 373.1197
## TIME_TO_DOM_INTERACTIVE_MS 2131.7333 1790.2477 341.4856
## TIME_TO_LOAD_EVENT_END_MS 3572.6793 2980.7217 591.9576
## TIME_TO_NON_BLANK_PAINT_MS 1693.1425 1469.7877 223.3548
## eQQ Med eQQ Mean eQQ Max
## distance 0.0091 0.0085 0.0133
## daily_num_sessions_started 0.1429 0.1641 5.0833
## daily_num_sessions_started_max 0.0000 0.2424 8.0000
## FX_PAGE_LOAD_MS_2_PARENT 213.4000 302.3883 731.4039
## memory_mb 48.0000 702.4733 65508.0000
## num_active_days 0.0000 0.1438 1.0000
## num_addons 0.1667 0.3024 1.3333
## num_bookmarks 12.0000 55.1384 12400.0000
## profile_age 36.0000 38.5212 1230.0000
## session_length 1.0283 1.4521 60.5250
## session_length_max 2.3419 3.0008 400.1297
## TIME_TO_DOM_COMPLETE_MS 245.0491 618.3153 6224.5733
## TIME_TO_DOM_CONTENT_LOADED_END_MS 190.9594 373.4239 7780.7182
## TIME_TO_DOM_INTERACTIVE_MS 163.5461 341.6326 5029.5595
## TIME_TO_LOAD_EVENT_END_MS 244.5388 592.0556 6676.2247
## TIME_TO_NON_BLANK_PAINT_MS 116.2831 223.1503 4474.0833
##
## , , Subclass 3
##
## Means Treated Means Control Mean Diff
## distance 0.6414 0.6312 0.0102
## daily_num_sessions_started 2.5052 3.3211 -0.8159
## daily_num_sessions_started_max 4.4751 6.1976 -1.7225
## FX_PAGE_LOAD_MS_2_PARENT 3082.7227 2697.5427 385.1800
## memory_mb 8283.3329 9780.1111 -1496.7782
## num_active_days 5.2337 6.1698 -0.9361
## num_addons 5.3065 5.9614 -0.6548
## num_bookmarks 116.6804 180.8574 -64.1770
## profile_age 868.5729 981.4336 -112.8606
## session_length 9.6627 6.6615 3.0012
## session_length_max 18.7577 14.3375 4.4201
## TIME_TO_DOM_COMPLETE_MS 3286.4351 2601.3614 685.0736
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2200.7573 2018.2117 182.5456
## TIME_TO_DOM_INTERACTIVE_MS 1808.2958 1398.2145 410.0814
## TIME_TO_LOAD_EVENT_END_MS 3041.4807 2324.6774 716.8033
## TIME_TO_NON_BLANK_PAINT_MS 1458.5905 1195.1818 263.4087
## eQQ Med eQQ Mean eQQ Max
## distance 0.0118 0.0102 0.0152
## daily_num_sessions_started 0.4286 0.8174 9.0000
## daily_num_sessions_started_max 1.0000 1.7274 16.0000
## FX_PAGE_LOAD_MS_2_PARENT 338.5206 387.0891 1085.0500
## memory_mb 77.0000 1496.5243 33966.0000
## num_active_days 1.0000 0.9372 2.0000
## num_addons 0.8333 0.6633 3.0000
## num_bookmarks 1.0000 68.2776 21670.0000
## profile_age 122.0000 114.9576 2620.0000
## session_length 2.4198 3.0087 15.3549
## session_length_max 3.4740 4.5009 137.5303
## TIME_TO_DOM_COMPLETE_MS 405.7366 684.6908 14361.0881
## TIME_TO_DOM_CONTENT_LOADED_END_MS 232.4408 378.2134 7292.1079
## TIME_TO_DOM_INTERACTIVE_MS 236.0310 411.0909 9502.8585
## TIME_TO_LOAD_EVENT_END_MS 435.4809 719.0484 5300.0548
## TIME_TO_NON_BLANK_PAINT_MS 170.1881 278.8717 8897.0500
##
## , , Subclass 4
##
## Means Treated Means Control Mean Diff
## distance 0.7105 0.7036 0.0070
## daily_num_sessions_started 2.6351 4.9025 -2.2675
## daily_num_sessions_started_max 4.6979 9.2742 -4.5763
## FX_PAGE_LOAD_MS_2_PARENT 2896.2967 2723.3355 172.9612
## memory_mb 8547.0695 13098.3097 -4551.2402
## num_active_days 5.2882 6.6333 -1.3451
## num_addons 4.7944 5.7903 -0.9960
## num_bookmarks 105.4028 404.5434 -299.1406
## profile_age 872.5596 1077.3452 -204.7855
## session_length 8.5216 5.7600 2.7616
## session_length_max 16.9851 14.1520 2.8331
## TIME_TO_DOM_COMPLETE_MS 2945.9427 2625.1679 320.7748
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2007.8868 2625.5060 -617.6193
## TIME_TO_DOM_INTERACTIVE_MS 1617.0368 1369.8993 247.1375
## TIME_TO_LOAD_EVENT_END_MS 2705.7776 2182.2139 523.5638
## TIME_TO_NON_BLANK_PAINT_MS 1330.7381 1164.8446 165.8935
## eQQ Med eQQ Mean eQQ Max
## distance 0.0075 0.0070 0.0112
## daily_num_sessions_started 1.3750 2.2555 11.2500
## daily_num_sessions_started_max 3.0000 4.5914 30.0000
## FX_PAGE_LOAD_MS_2_PARENT 181.5297 195.7069 853.9716
## memory_mb 2029.5000 4547.7430 65589.0000
## num_active_days 2.0000 1.3462 3.0000
## num_addons 1.0000 1.0063 3.0000
## num_bookmarks 11.0000 293.8034 20326.3000
## profile_age 221.0000 203.5376 1370.0000
## session_length 2.7746 3.1834 140.2416
## session_length_max 3.4928 6.1577 963.7425
## TIME_TO_DOM_COMPLETE_MS 335.7212 414.9658 11959.6875
## TIME_TO_DOM_CONTENT_LOADED_END_MS 102.6898 741.2994 15199.5967
## TIME_TO_DOM_INTERACTIVE_MS 243.2424 321.6764 15693.5000
## TIME_TO_LOAD_EVENT_END_MS 390.9676 563.0875 4736.1065
## TIME_TO_NON_BLANK_PAINT_MS 153.9446 227.6094 13678.3571
##
## , , Subclass 5
##
## Means Treated Means Control Mean Diff
## distance 0.7648 0.7607 0.0041
## daily_num_sessions_started 2.9024 5.7899 -2.8874
## daily_num_sessions_started_max 5.2491 10.7635 -5.5144
## FX_PAGE_LOAD_MS_2_PARENT 2666.1675 2609.6019 56.5656
## memory_mb 9107.3124 20377.8133 -11270.5009
## num_active_days 5.6015 6.4191 -0.8176
## num_addons 4.4209 5.5429 -1.1220
## num_bookmarks 112.2892 592.6343 -480.3451
## profile_age 861.0621 980.2988 -119.2367
## session_length 7.5347 5.1380 2.3968
## session_length_max 15.2823 11.1722 4.1101
## TIME_TO_DOM_COMPLETE_MS 2625.0234 2780.4329 -155.4096
## TIME_TO_DOM_CONTENT_LOADED_END_MS 1870.5593 3509.5796 -1639.0203
## TIME_TO_DOM_INTERACTIVE_MS 1419.4107 1432.1830 -12.7723
## TIME_TO_LOAD_EVENT_END_MS 2368.0543 2238.5444 129.5099
## TIME_TO_NON_BLANK_PAINT_MS 1189.4232 1187.2014 2.2218
## eQQ Med eQQ Mean eQQ Max
## distance 0.0045 0.0041 0.0072
## daily_num_sessions_started 1.3000 2.8410 13.7083
## daily_num_sessions_started_max 3.0000 5.4315 39.0000
## FX_PAGE_LOAD_MS_2_PARENT 95.0470 121.8725 826.9945
## memory_mb 8183.0000 11399.3195 57277.0000
## num_active_days 1.0000 0.8340 2.0000
## num_addons 1.0000 1.1622 2.0000
## num_bookmarks 14.0000 430.0617 18798.0000
## profile_age 111.0000 112.0373 856.0000
## session_length 2.2544 2.5946 25.4690
## session_length_max 3.4842 5.3169 211.4811
## TIME_TO_DOM_COMPLETE_MS 187.9138 441.8738 31221.1000
## TIME_TO_DOM_CONTENT_LOADED_END_MS 66.5836 1637.8773 17465.4796
## TIME_TO_DOM_INTERACTIVE_MS 127.3965 224.3370 6019.0057
## TIME_TO_LOAD_EVENT_END_MS 279.2186 341.0763 5009.2917
## TIME_TO_NON_BLANK_PAINT_MS 112.7420 393.1289 49277.3333
##
## , , Subclass 6
##
## Means Treated Means Control Mean Diff
## distance 0.8272 0.8505 -0.0233
## daily_num_sessions_started 4.3318 4.9130 -0.5812
## daily_num_sessions_started_max 8.0604 9.6513 -1.5909
## FX_PAGE_LOAD_MS_2_PARENT 2632.2594 2751.3244 -119.0650
## memory_mb 12686.3512 30182.8421 -17496.4909
## num_active_days 6.4459 5.7697 0.6761
## num_addons 4.1663 4.5669 -0.4006
## num_bookmarks 179.2124 2237.2700 -2058.0576
## profile_age 923.5093 896.8355 26.6738
## session_length 5.8352 9.0870 -3.2518
## session_length_max 12.8894 28.3594 -15.4700
## TIME_TO_DOM_COMPLETE_MS 2475.6838 2774.1794 -298.4956
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2219.2748 3771.2138 -1551.9390
## TIME_TO_DOM_INTERACTIVE_MS 1263.4862 1659.4946 -396.0083
## TIME_TO_LOAD_EVENT_END_MS 2101.5986 2478.4162 -376.8176
## TIME_TO_NON_BLANK_PAINT_MS 1085.8839 1442.3090 -356.4251
## eQQ Med eQQ Mean eQQ Max
## distance 0.0169 0.0231 0.0810
## daily_num_sessions_started 0.4375 0.9971 7.5714
## daily_num_sessions_started_max 1.0000 2.5724 29.0000
## FX_PAGE_LOAD_MS_2_PARENT 99.6082 187.1169 1715.6173
## memory_mb 1405.5000 18303.7961 196547.0000
## num_active_days 1.0000 0.6711 2.0000
## num_addons 1.6333 1.6563 4.0000
## num_bookmarks 5.0000 1982.8786 31046.4286
## profile_age 34.0000 83.9211 2559.0000
## session_length 0.4472 2.9981 121.9651
## session_length_max 0.7782 13.6103 1033.4492
## TIME_TO_DOM_COMPLETE_MS 95.0598 458.3183 13187.8061
## TIME_TO_DOM_CONTENT_LOADED_END_MS 123.3698 1429.7080 11419.9288
## TIME_TO_DOM_INTERACTIVE_MS 50.9078 360.5745 7145.0000
## TIME_TO_LOAD_EVENT_END_MS 52.9427 404.1767 4622.9001
## TIME_TO_NON_BLANK_PAINT_MS 43.7286 741.9514 74840.5000
##
##
## Sample sizes by subclasses:
## Subclass 1 Subclass 2 Subclass 3 Subclass 4 Subclass 5 Subclass 6
## Treated 5829 5829 5828 5829 5829 5829
## Control 19144 10932 3628 930 241 152
## Total 24973 16761 9456 6759 6070 5981
##
## Summary of balance across subclasses
## Means Treated Means Control Mean Diff
## distance 0.6201 0.6153 0.0060
## daily_num_sessions_started 2.8624 3.8864 0.6368
## daily_num_sessions_started_max 5.1994 7.2888 1.2628
## FX_PAGE_LOAD_MS_2_PARENT 3044.1381 2954.0437 99.0162
## memory_mb 9375.4970 15048.6562 3560.7322
## num_active_days 5.5615 5.9116 0.3354
## num_addons 5.6684 6.2448 0.2833
## num_bookmarks 164.7282 639.4241 356.0054
## profile_age 899.2399 949.3047 46.0377
## session_length 9.2079 8.5845 1.0770
## session_length_max 18.1451 18.9420 2.9032
## TIME_TO_DOM_COMPLETE_MS 3304.4993 3255.1950 225.5580
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2291.1852 2879.1307 398.7701
## TIME_TO_DOM_INTERACTIVE_MS 1797.9580 1779.5984 142.6862
## TIME_TO_LOAD_EVENT_END_MS 3026.7281 2912.1305 241.6520
## TIME_TO_NON_BLANK_PAINT_MS 1457.4383 1445.4678 95.0399
## eQQ Med eQQ Mean eQQ Max
## distance 0.0121 0.0126 0.0282
## daily_num_sessions_started 0.6696 1.2386 8.7272
## daily_num_sessions_started_max 1.5000 2.5565 25.3336
## FX_PAGE_LOAD_MS_2_PARENT 186.1789 245.7397 990.0750
## memory_mb 1963.8873 6177.8073 75276.5145
## num_active_days 0.8333 0.7331 1.8333
## num_addons 0.8278 0.8807 23.3339
## num_bookmarks 10.8336 480.5461 21111.1054
## profile_age 100.3327 104.6571 1644.3054
## session_length 1.7847 2.6410 85.7967
## session_length_max 2.5476 6.0231 555.4638
## TIME_TO_DOM_COMPLETE_MS 277.6407 584.4370 13945.6388
## TIME_TO_DOM_CONTENT_LOADED_END_MS 141.1896 806.8060 13190.0405
## TIME_TO_DOM_INTERACTIVE_MS 169.4842 357.5374 11034.0866
## TIME_TO_LOAD_EVENT_END_MS 302.6421 587.8601 5510.7104
## TIME_TO_NON_BLANK_PAINT_MS 110.3260 350.6535 33520.1152
##
## Percent Balance Improvement:
## Mean Diff. eQQ Med eQQ Mean eQQ Max
## distance 97.9984 95.1456 94.7853 90.9244
## daily_num_sessions_started -116.8928 -67.3876 -161.4850 -454.1063
## daily_num_sessions_started_max -135.9096 -50.0014 -187.8414 -744.4533
## FX_PAGE_LOAD_MS_2_PARENT 78.3081 34.9827 40.8215 18.6482
## memory_mb -1246.8861 -6235.1203 -1039.5335 61.7005
## num_active_days -59.7164 -Inf -208.8670 -83.3329
## num_addons 73.7930 58.6111 59.9566 81.5785
## num_bookmarks -498.1511 -983.3615 -507.0711 3.0222
## profile_age -768.4716 -336.2292 -313.8720 -20.1100
## session_length 79.1762 -22.5368 11.6827 43.2609
## session_length_max 81.9806 11.1142 -36.7763 35.2870
## TIME_TO_DOM_COMPLETE_MS 95.4895 32.0805 46.4872 -20.8581
## TIME_TO_DOM_CONTENT_LOADED_END_MS -29.9490 35.3306 -78.6990 -6.0336
## TIME_TO_DOM_INTERACTIVE_MS 97.0084 28.1872 41.6373 46.4547
## TIME_TO_LOAD_EVENT_END_MS 89.6547 29.3429 46.8820 50.7806
## TIME_TO_NON_BLANK_PAINT_MS 96.8515 27.1956 7.4922 -108.1668Match using Subclassification matching: Full Dataset
sub_full <- matchit(formula = generate_formula(exp_3, label), df_1x, 'subclass')
res <- match.data(sub_full)
sub_full_models_1x <- sub_full
sub_full_results_1x <- res
print(summary(sub_full))##
## Call:
## matchit(formula = generate_formula(exp_3, label), data = df_1x,
## method = "subclass", sub.by = "treat")
## Summary of balance for all data:
## Means Treated Means Control Mean Diff
## distance 0.6213 0.3787 0.2426
## daily_num_sessions_started 2.8876 2.3689 0.5188
## daily_num_sessions_started_max 5.2501 4.2814 0.9687
## FX_PAGE_LOAD_MS_2_PARENT 3027.5228 3463.7084 -436.1856
## memory_mb 9460.0891 8965.1561 494.9330
## num_active_days 5.5746 5.3462 0.2284
## num_addons 5.6632 7.8554 -2.1922
## num_bookmarks 164.1029 242.4878 -78.3849
## profile_age 893.9339 893.7534 0.1805
## session_length 9.2495 12.2962 -3.0467
## session_length_max 18.1985 22.7066 -4.5080
## TIME_TO_DOM_COMPLETE_MS 3285.4718 4388.9143 -1103.4424
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2292.5533 2737.6381 -445.0848
## TIME_TO_DOM_INTERACTIVE_MS 1792.4896 2404.3932 -611.9036
## TIME_TO_LOAD_EVENT_END_MS 3012.8713 4126.9123 -1114.0410
## TIME_TO_NON_BLANK_PAINT_MS 1449.9423 1833.6557 -383.7134
## eQQ Med eQQ Mean eQQ Max
## distance 0.2527 0.2426 0.3114
## daily_num_sessions_started 0.4167 0.5190 1.7500
## daily_num_sessions_started_max 1.0000 0.9692 11.0000
## FX_PAGE_LOAD_MS_2_PARENT 299.6662 436.2548 1250.9746
## memory_mb 31.0000 555.3761 262176.0000
## num_active_days 0.0000 0.2413 1.0000
## num_addons 2.0000 2.1941 109.0000
## num_bookmarks 1.0000 79.0181 21769.0000
## profile_age 22.0000 23.1161 1366.0000
## session_length 1.4538 3.0489 151.4597
## session_length_max 2.7333 4.5187 871.0939
## TIME_TO_DOM_COMPLETE_MS 421.7080 1103.4946 12182.3743
## TIME_TO_DOM_CONTENT_LOADED_END_MS 229.0178 445.1832 10634.8539
## TIME_TO_DOM_INTERACTIVE_MS 243.7872 612.0582 19062.0000
## TIME_TO_LOAD_EVENT_END_MS 441.3178 1114.0691 11196.2118
## TIME_TO_NON_BLANK_PAINT_MS 155.3258 384.1746 19260.1667
##
##
## Summary of balance by subclasses:
## , , Subclass 1
##
## Means Treated Means Control Mean Diff
## distance 0.2516 0.2302 0.0214
## daily_num_sessions_started 2.4101 2.0198 0.3903
## daily_num_sessions_started_max 4.4171 3.6037 0.8134
## FX_PAGE_LOAD_MS_2_PARENT 3645.6022 3894.3402 -248.7380
## memory_mb 9111.6550 8806.4754 305.1795
## num_active_days 5.5209 5.0730 0.4480
## num_addons 9.1570 9.1982 -0.0412
## num_bookmarks 305.6801 287.6036 18.0765
## profile_age 942.1902 872.1816 70.0086
## session_length 12.3416 15.1584 -2.8168
## session_length_max 23.1106 26.8892 -3.7786
## TIME_TO_DOM_COMPLETE_MS 4672.6893 5493.3005 -820.6112
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2969.6198 3189.9871 -220.3673
## TIME_TO_DOM_INTERACTIVE_MS 2563.9265 2997.3077 -433.3811
## TIME_TO_LOAD_EVENT_END_MS 4383.9143 5234.3483 -850.4340
## TIME_TO_NON_BLANK_PAINT_MS 2013.0789 2193.1474 -180.0685
## eQQ Med eQQ Mean eQQ Max
## distance 0.0223 0.0214 0.0368
## daily_num_sessions_started 0.3393 0.3917 2.6250
## daily_num_sessions_started_max 1.0000 0.8153 11.0000
## FX_PAGE_LOAD_MS_2_PARENT 164.4150 273.4149 691.8043
## memory_mb 34.0000 558.1491 65529.0000
## num_active_days 0.0000 0.4480 1.0000
## num_addons 0.3333 0.4836 109.0000
## num_bookmarks 22.0000 56.0556 22426.0000
## profile_age 72.0000 71.1095 1228.0000
## session_length 1.8137 2.8224 151.2126
## session_length_max 1.5853 3.8216 586.3783
## TIME_TO_DOM_COMPLETE_MS 348.9241 834.4865 3311.9972
## TIME_TO_DOM_CONTENT_LOADED_END_MS 93.7953 244.5282 15414.5714
## TIME_TO_DOM_INTERACTIVE_MS 167.5294 446.8677 18812.3581
## TIME_TO_LOAD_EVENT_END_MS 369.7923 860.9821 3421.6268
## TIME_TO_NON_BLANK_PAINT_MS 47.9215 197.2894 41949.0000
##
## , , Subclass 2
##
## Means Treated Means Control Mean Diff
## distance 0.5271 0.5179 0.0092
## daily_num_sessions_started 2.3838 2.3032 0.0807
## daily_num_sessions_started_max 4.2884 4.1319 0.1565
## FX_PAGE_LOAD_MS_2_PARENT 3354.8414 3054.6816 300.1598
## memory_mb 8518.6848 8010.2872 508.3976
## num_active_days 5.2812 5.4200 -0.1387
## num_addons 6.1290 6.3419 -0.2129
## num_bookmarks 168.1025 136.4346 31.6679
## profile_age 924.5122 898.0909 26.4212
## session_length 11.2709 9.8223 1.4486
## session_length_max 21.6102 19.0209 2.5893
## TIME_TO_DOM_COMPLETE_MS 3822.8501 3219.8669 602.9832
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2476.8607 2140.6634 336.1973
## TIME_TO_DOM_INTERACTIVE_MS 2111.5677 1793.6102 317.9575
## TIME_TO_LOAD_EVENT_END_MS 3556.3707 2979.4544 576.9163
## TIME_TO_NON_BLANK_PAINT_MS 1673.8577 1471.3043 202.5534
## eQQ Med eQQ Mean eQQ Max
## distance 0.0100 0.0092 0.0139
## daily_num_sessions_started 0.1250 0.1477 4.3750
## daily_num_sessions_started_max 0.0000 0.2282 10.0000
## FX_PAGE_LOAD_MS_2_PARENT 222.0587 300.3449 722.5508
## memory_mb 42.0000 621.9851 65458.0000
## num_active_days 0.0000 0.1386 1.0000
## num_addons 0.1667 0.3129 1.5000
## num_bookmarks 12.1429 47.0343 14227.0000
## profile_age 24.0000 27.3316 492.0000
## session_length 1.0825 1.4915 100.8215
## session_length_max 2.2539 3.0444 992.3211
## TIME_TO_DOM_COMPLETE_MS 243.5112 603.2329 9796.1429
## TIME_TO_DOM_CONTENT_LOADED_END_MS 169.5215 337.2614 8263.6343
## TIME_TO_DOM_INTERACTIVE_MS 146.1880 318.0017 8589.9028
## TIME_TO_LOAD_EVENT_END_MS 232.4506 576.8566 6005.7348
## TIME_TO_NON_BLANK_PAINT_MS 110.2811 202.7836 4541.0000
##
## , , Subclass 3
##
## Means Treated Means Control Mean Diff
## distance 0.6420 0.6325 0.0094
## daily_num_sessions_started 2.4619 3.3168 -0.8549
## daily_num_sessions_started_max 4.4122 6.1918 -1.7796
## FX_PAGE_LOAD_MS_2_PARENT 3086.4699 2659.3165 427.1534
## memory_mb 8344.9531 9961.1075 -1616.1544
## num_active_days 5.2810 6.1962 -0.9152
## num_addons 5.2979 5.9753 -0.6774
## num_bookmarks 121.6145 191.1804 -69.5658
## profile_age 876.7948 971.0145 -94.2197
## session_length 9.8206 6.6121 3.2085
## session_length_max 19.0226 14.4456 4.5770
## TIME_TO_DOM_COMPLETE_MS 3284.0538 2580.3769 703.6769
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2193.2978 2004.0599 189.2378
## TIME_TO_DOM_INTERACTIVE_MS 1807.3320 1400.5442 406.7879
## TIME_TO_LOAD_EVENT_END_MS 3043.2840 2291.9163 751.3678
## TIME_TO_NON_BLANK_PAINT_MS 1460.3143 1202.5560 257.7584
## eQQ Med eQQ Mean eQQ Max
## distance 0.0108 0.0094 0.0140
## daily_num_sessions_started 0.5250 0.8557 7.5500
## daily_num_sessions_started_max 1.0000 1.7844 29.0000
## FX_PAGE_LOAD_MS_2_PARENT 366.1511 428.3443 1136.7030
## memory_mb 75.0000 1617.7099 32761.0000
## num_active_days 1.0000 0.9157 2.0000
## num_addons 0.9167 0.6798 2.4545
## num_bookmarks 1.0000 75.2084 19765.0000
## profile_age 102.0000 98.8166 2052.0000
## session_length 2.7476 3.2756 123.8998
## session_length_max 4.2128 5.2575 995.9783
## TIME_TO_DOM_COMPLETE_MS 440.2364 704.7485 3050.4283
## TIME_TO_DOM_CONTENT_LOADED_END_MS 245.2150 371.2476 5581.5286
## TIME_TO_DOM_INTERACTIVE_MS 253.5394 411.1351 3351.0788
## TIME_TO_LOAD_EVENT_END_MS 465.0222 752.1617 3870.4200
## TIME_TO_NON_BLANK_PAINT_MS 178.8468 300.0734 37901.1429
##
## , , Subclass 4
##
## Means Treated Means Control Mean Diff
## distance 0.7114 0.7050 0.0064
## daily_num_sessions_started 2.6317 5.0880 -2.4563
## daily_num_sessions_started_max 4.7066 9.6201 -4.9135
## FX_PAGE_LOAD_MS_2_PARENT 2899.4335 2666.3722 233.0612
## memory_mb 8514.6531 13264.7341 -4750.0809
## num_active_days 5.3143 6.5885 -1.2742
## num_addons 4.7867 5.7945 -1.0078
## num_bookmarks 103.0137 283.8290 -180.8153
## profile_age 864.8687 999.2260 -134.3573
## session_length 8.6304 5.5841 3.0463
## session_length_max 17.3249 12.8122 4.5126
## TIME_TO_DOM_COMPLETE_MS 2945.8132 2649.6120 296.2013
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2025.2565 2654.9082 -629.6517
## TIME_TO_DOM_INTERACTIVE_MS 1617.3892 1370.7463 246.6429
## TIME_TO_LOAD_EVENT_END_MS 2699.1145 2191.5727 507.5417
## TIME_TO_NON_BLANK_PAINT_MS 1324.0560 1157.6885 166.3676
## eQQ Med eQQ Mean eQQ Max
## distance 0.0068 0.0064 0.0103
## daily_num_sessions_started 1.7143 2.4516 11.2500
## daily_num_sessions_started_max 4.0000 4.9041 30.0000
## FX_PAGE_LOAD_MS_2_PARENT 239.0448 246.9080 764.8333
## memory_mb 2248.0000 4722.5969 33966.0000
## num_active_days 1.0000 1.2750 2.0000
## num_addons 1.0000 1.0189 3.2000
## num_bookmarks 11.0000 175.9415 14549.0000
## profile_age 149.0000 135.4804 1604.0000
## session_length 2.8483 3.0998 15.5721
## session_length_max 3.5917 5.2588 209.0297
## TIME_TO_DOM_COMPLETE_MS 325.4478 449.0558 23673.0966
## TIME_TO_DOM_CONTENT_LOADED_END_MS 112.6439 777.1600 11550.4087
## TIME_TO_DOM_INTERACTIVE_MS 246.9167 331.1044 13564.9171
## TIME_TO_LOAD_EVENT_END_MS 391.9440 552.3713 8816.9173
## TIME_TO_NON_BLANK_PAINT_MS 168.9755 242.1909 10601.6000
##
## , , Subclass 5
##
## Means Treated Means Control Mean Diff
## distance 0.7657 0.7617 0.0040
## daily_num_sessions_started 2.8739 6.3624 -3.4885
## daily_num_sessions_started_max 5.2190 11.7263 -6.5073
## FX_PAGE_LOAD_MS_2_PARENT 2633.8482 2562.6295 71.2187
## memory_mb 9125.3363 19293.1744 -10167.8381
## num_active_days 5.6194 6.5342 -0.9148
## num_addons 4.4228 5.6194 -1.1966
## num_bookmarks 109.5189 453.3226 -343.8037
## profile_age 855.5199 940.0817 -84.5618
## session_length 7.6116 5.1372 2.4744
## session_length_max 15.2832 11.1064 4.1768
## TIME_TO_DOM_COMPLETE_MS 2583.9345 2466.1432 117.7913
## TIME_TO_DOM_CONTENT_LOADED_END_MS 1878.4901 3370.1628 -1491.6727
## TIME_TO_DOM_INTERACTIVE_MS 1409.5434 1378.0845 31.4589
## TIME_TO_LOAD_EVENT_END_MS 2331.8125 2111.0402 220.7723
## TIME_TO_NON_BLANK_PAINT_MS 1174.0838 1256.6035 -82.5197
## eQQ Med eQQ Mean eQQ Max
## distance 0.0043 0.0040 0.0062
## daily_num_sessions_started 1.5833 3.4714 13.7083
## daily_num_sessions_started_max 3.0000 6.4901 23.0000
## FX_PAGE_LOAD_MS_2_PARENT 121.7924 164.3502 1622.7155
## memory_mb 7495.0000 9993.7042 49113.0000
## num_active_days 1.0000 0.9249 2.0000
## num_addons 1.0000 1.2433 2.6667
## num_bookmarks 11.7500 317.9802 18798.0000
## profile_age 87.0000 83.0508 1351.0000
## session_length 2.1457 2.5583 25.4151
## session_length_max 2.8511 4.8221 169.2850
## TIME_TO_DOM_COMPLETE_MS 257.8807 354.0923 10297.8289
## TIME_TO_DOM_CONTENT_LOADED_END_MS 85.9501 1537.6741 14560.9670
## TIME_TO_DOM_INTERACTIVE_MS 157.4279 239.3562 6359.4583
## TIME_TO_LOAD_EVENT_END_MS 300.3357 379.8884 4270.1067
## TIME_TO_NON_BLANK_PAINT_MS 118.9085 266.0568 28137.7476
##
## , , Subclass 6
##
## Means Treated Means Control Mean Diff
## distance 0.8300 0.8482 -0.0182
## daily_num_sessions_started 4.5646 6.0655 -1.5010
## daily_num_sessions_started_max 8.4573 11.5828 -3.1254
## FX_PAGE_LOAD_MS_2_PARENT 2544.9476 2675.1385 -130.1909
## memory_mb 13145.1402 27851.7517 -14706.6115
## num_active_days 6.4306 5.9139 0.5167
## num_addons 4.1856 4.7301 -0.5444
## num_bookmarks 176.6831 2294.1172 -2117.4340
## profile_age 899.7160 866.1954 33.5207
## session_length 5.8220 7.6336 -1.8116
## session_length_max 12.8399 21.4820 -8.6421
## TIME_TO_DOM_COMPLETE_MS 2403.4901 2629.9196 -226.4295
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2211.7849 3753.0673 -1541.2824
## TIME_TO_DOM_INTERACTIVE_MS 1245.1803 1571.1946 -326.0142
## TIME_TO_LOAD_EVENT_END_MS 2062.7350 2373.7297 -310.9947
## TIME_TO_NON_BLANK_PAINT_MS 1054.2640 1461.4806 -407.2166
## eQQ Med eQQ Mean eQQ Max
## distance 0.0068 0.0184 0.0751
## daily_num_sessions_started 0.3429 1.8072 11.0000
## daily_num_sessions_started_max 1.0000 3.7781 25.0000
## FX_PAGE_LOAD_MS_2_PARENT 70.3052 183.7392 1999.6909
## memory_mb 1938.0000 15460.0099 262176.0000
## num_active_days 0.0000 0.5132 2.0000
## num_addons 1.6667 1.6546 4.0000
## num_bookmarks 2.0000 2085.3162 32271.0417
## profile_age 39.0000 77.8344 2274.0000
## session_length 0.1678 1.8351 119.4322
## session_length_max 0.3888 8.3654 1021.1394
## TIME_TO_DOM_COMPLETE_MS 103.8242 374.9957 13187.8061
## TIME_TO_DOM_CONTENT_LOADED_END_MS 156.2572 1501.2906 12944.3276
## TIME_TO_DOM_INTERACTIVE_MS 49.7759 308.9210 13466.7365
## TIME_TO_LOAD_EVENT_END_MS 91.5503 390.9446 4908.1131
## TIME_TO_NON_BLANK_PAINT_MS 38.8486 518.6575 59385.5000
##
##
## Sample sizes by subclasses:
## Subclass 1 Subclass 2 Subclass 3 Subclass 4 Subclass 5 Subclass 6
## Treated 9938 9938 9937 9938 9938 9938
## Control 32931 18517 5871 1553 453 302
## Total 42869 28455 15808 11491 10391 10240
##
## Summary of balance across subclasses
## Means Treated Means Control Mean Diff
## distance 0.6213 0.6159 0.0053
## daily_num_sessions_started 2.8876 4.1926 0.7700
## daily_num_sessions_started_max 5.2501 7.8095 1.4917
## FX_PAGE_LOAD_MS_2_PARENT 3027.5228 2918.7508 106.8154
## memory_mb 9460.0891 14531.3317 3096.6255
## num_active_days 5.5746 5.9543 0.3242
## num_addons 5.6632 6.2766 0.3005
## num_bookmarks 164.1029 607.7549 359.0398
## profile_age 893.9339 924.4642 33.6660
## session_length 9.2495 8.3247 1.0410
## session_length_max 18.1985 17.6261 2.0711
## TIME_TO_DOM_COMPLETE_MS 3285.4718 3173.2131 216.3453
## TIME_TO_DOM_CONTENT_LOADED_END_MS 2292.5533 2852.1557 379.8636
## TIME_TO_DOM_INTERACTIVE_MS 1792.4896 1751.9205 131.4965
## TIME_TO_LOAD_EVENT_END_MS 3012.8713 2863.6865 237.0909
## TIME_TO_NON_BLANK_PAINT_MS 1449.9423 1457.1343 97.2115
## eQQ Med eQQ Mean eQQ Max
## distance 0.0102 0.0115 0.0260
## daily_num_sessions_started 0.7716 1.5209 8.4181
## daily_num_sessions_started_max 1.6667 3.0000 21.3332
## FX_PAGE_LOAD_MS_2_PARENT 197.2917 266.1809 1156.3833
## memory_mb 1972.0318 5495.7576 84834.7066
## num_active_days 0.5000 0.7026 1.6667
## num_addons 0.8472 0.8988 20.4705
## num_bookmarks 9.9823 459.5958 20339.3499
## profile_age 78.8329 82.2702 1500.1574
## session_length 1.8009 2.5138 89.3916
## session_length_max 2.4805 5.0950 662.3497
## TIME_TO_DOM_COMPLETE_MS 286.6349 553.4327 10553.0092
## TIME_TO_DOM_CONTENT_LOADED_END_MS 143.8955 794.8674 11386.0036
## TIME_TO_DOM_INTERACTIVE_MS 170.2282 342.5632 10690.8650
## TIME_TO_LOAD_EVENT_END_MS 308.5132 585.5313 5215.5090
## TIME_TO_NON_BLANK_PAINT_MS 110.6292 287.8417 30419.2063
##
## Percent Balance Improvement:
## Mean Diff. eQQ Med eQQ Mean eQQ Max
## distance 97.7885 95.9832 95.2700 91.6337
## daily_num_sessions_started -151.5586 -85.1915 -193.0205 -381.0326
## daily_num_sessions_started_max -164.2034 -66.6678 -209.5455 -93.9382
## FX_PAGE_LOAD_MS_2_PARENT 75.0629 34.1628 38.9850 7.5614
## memory_mb -924.6321 -6261.3929 -889.5560 67.6421
## num_active_days -66.2352 -Inf -191.1955 -66.6661
## num_addons 72.0196 57.6389 59.0349 81.2197
## num_bookmarks -465.9914 -898.2293 -481.6335 6.5674
## profile_age -16810.6400 -258.3316 -255.9004 -9.8212
## session_length 69.6434 -23.8817 17.5515 40.9799
## session_length_max 87.3018 9.2482 -12.7543 23.9635
## TIME_TO_DOM_COMPLETE_MS 89.8265 32.0300 49.8473 13.3748
## TIME_TO_DOM_CONTENT_LOADED_END_MS -25.7294 37.1684 -78.5484 -7.0631
## TIME_TO_DOM_INTERACTIVE_MS 93.3700 30.1735 44.0309 43.9153
## TIME_TO_LOAD_EVENT_END_MS 86.6087 30.0927 47.4421 53.4172
## TIME_TO_NON_BLANK_PAINT_MS 98.1257 28.7760 25.0753 -57.9384Graphical Comparison
small <- list(cem_models_1x = cem_models_1x, nn_models_1x = nn_models_1x, nn.ma_models_1x = nn.ma_models_1x, sub_models_1x = sub_models_1x)
full <- list(cem_full_models_1x = cem_full_models_1x, nn_full_models_1x = nn_full_models_1x, nn.ma_full_models_1x = nn.ma_full_models_1x, sub_full_models_1x = sub_full_models_1x)- Note that we want to see if the Matched Treated (beta) and Matched Control (release) distributions are roughly similar.
Small Dataset
for (model in names(small)){
print(paste(model))
if (model != "nn.ma_models_1x"){
plot(small[[model]], type = "hist", col = "#111d5e")
plot(small[[model]], type = "jitter", col = "#111d5e", interactive = F)
} else {
print(paste("Not appropriate for pure Mahalanobis matching. No plots generated."))
}
}## [1] "cem_models_1x"
## [1] "nn_models_1x"
## [1] "nn.ma_models_1x"
## [1] "Not appropriate for pure Mahalanobis matching. No plots generated."
## [1] "sub_models_1x"
Full Dataset
for (model in names(full)){
print(paste(model))
if (model != "nn.ma_full_models_1x"){
plot(full[[model]], type = "hist", col = "#111d5e")
plot(full[[model]], type = "jitter", col = "#111d5e", interactive = F)
} else {
print(paste("Not appropriate for pure Mahalanobis matching. No plots generated."))
}
}## [1] "cem_full_models_1x"
## [1] "nn_full_models_1x"
## [1] "nn.ma_full_models_1x"
## [1] "Not appropriate for pure Mahalanobis matching. No plots generated."
## [1] "sub_full_models_1x"
Observations
- Although it was not possible to graphically compare the results using the Nearest Neighbor matching with Mahalanobis distance, from the summary, we can see that those models outperformed the others. Analyzing each of the covariates, we have:
Small Dataset
for (v in exp_3){
p <- bal.plot(nn.ma_models_1x, var.name = v, colors = c("#111d5e", "#611e62"))
print(p)
}
Full Dataset
for (v in exp_3){
p <- bal.plot(nn.ma_full_models_1x, var.name = v, colors = c("#111d5e", "#611e62"))
print(p)
}
- The figures show that the model containing the entire dataset was better than sampling.
- In addition, the main problem is displayed mainly in pageload metrics.